Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dagster-fivetran] Implement base resync method in FivetranClient #26059

Merged

Conversation

maximearmstrong
Copy link
Contributor

@maximearmstrong maximearmstrong commented Nov 20, 2024

Summary & Motivation

This PR reworks legacy resync method and implements it in the FivetranClient:

Tests mock the request API calls and make sure that all calls are made.

How I Tested These Changes

Additional unit tests with BK

Copy link
Contributor Author

maximearmstrong commented Nov 20, 2024

connector_id (str): The Fivetran Connector ID. You can retrieve this value from the
"Setup" tab of a given connector in the Fivetran UI.
resync_parameters (Optional[Dict[str, List[str]]]): Optional resync parameters to send to the Fivetran API.
An example payload can be found here: https://fivetran.com/docs/rest-api/connectors#request_7
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome docstring

@maximearmstrong maximearmstrong force-pushed the maxime/rework-fivetran-9 branch from 8e607df to 6cd5f9e Compare November 20, 2024 23:24
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from cadbe37 to 90123af Compare November 20, 2024 23:24
@maximearmstrong maximearmstrong force-pushed the maxime/rework-fivetran-9 branch from 6cd5f9e to 8e3c5a9 Compare November 21, 2024 18:06
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from 90123af to b81d516 Compare November 21, 2024 18:06
@maximearmstrong maximearmstrong force-pushed the maxime/rework-fivetran-9 branch from 8e3c5a9 to 8e2d5cb Compare November 21, 2024 21:07
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from b81d516 to 004f258 Compare November 21, 2024 21:08
@maximearmstrong maximearmstrong force-pushed the maxime/rework-fivetran-9 branch from 8e2d5cb to f43b3f6 Compare November 21, 2024 23:45
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from 004f258 to 03b4f71 Compare November 21, 2024 23:45
@maximearmstrong maximearmstrong force-pushed the maxime/rework-fivetran-9 branch from f43b3f6 to f56783b Compare November 23, 2024 00:39
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from 03b4f71 to 6855c14 Compare November 23, 2024 00:39
@maximearmstrong maximearmstrong force-pushed the maxime/rework-fivetran-9 branch from f56783b to 72d546e Compare November 25, 2024 13:05
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from 6855c14 to 2021ba0 Compare November 25, 2024 13:06
@maximearmstrong maximearmstrong force-pushed the maxime/rework-fivetran-9 branch from 72d546e to a069876 Compare November 25, 2024 16:50
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from 2021ba0 to 3df2067 Compare November 25, 2024 16:50
@maximearmstrong maximearmstrong force-pushed the maxime/rework-fivetran-9 branch from a069876 to cec7700 Compare November 25, 2024 23:22
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from 3df2067 to 22bfe6c Compare November 25, 2024 23:22
@maximearmstrong maximearmstrong force-pushed the maxime/rework-fivetran-9 branch from cec7700 to 4e940a4 Compare November 26, 2024 22:49
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from 22bfe6c to ffab308 Compare November 26, 2024 22:49
@maximearmstrong maximearmstrong force-pushed the maxime/rework-fivetran-9 branch from 4e940a4 to 5381ecd Compare November 26, 2024 23:20
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from ffab308 to 16121e5 Compare November 26, 2024 23:21
Base automatically changed from maxime/rework-fivetran-9 to master November 27, 2024 13:27
@maximearmstrong maximearmstrong force-pushed the maxime/implement-resync-base-method-fivetran-client branch from 16121e5 to 8a48241 Compare November 27, 2024 13:31
@maximearmstrong maximearmstrong merged commit 0d85b0b into master Nov 27, 2024
1 check passed
@maximearmstrong maximearmstrong deleted the maxime/implement-resync-base-method-fivetran-client branch November 27, 2024 13:44
cmpadden pushed a commit that referenced this pull request Dec 5, 2024
…6059)

## Summary & Motivation

This PR reworks legacy resync method and implements it in the
`FivetranClient`:

-  `start_resync` is added based on legacy `start_resync`
- `start_resync` leverages `_start_sync` introduced in #25911 
- a [resync in
Fivetran](https://fivetran.com/docs/rest-api/api-reference/connectors/resync-connector)
is historical data sync - the endpoint and result is different, but
logic around how to call and handle a resync is the same as a sync.
- a resync can be done with or without resync parameters, using a
different endpoint.

Tests mock the request API calls and make sure that all calls are made.

## How I Tested These Changes

Additional unit tests with BK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants